Deep Reinforcement Learning with Feedback-based Exploration

Deep Reinforcement Learning has enabled the control of increasingly complex and high-dimensional problems. However, the need of vast amounts of data before reasonable performance is attained prevents its widespread application. We employ binary corrective feedback as a general and intuitive manner to incorporate human intuition and domain knowledge in model-free machine learning. The uncertainty in the policy and the corrective feedback is combined directly in the action space as probabilistic conditional exploration. As a result, the greatest part of the otherwise ignorant learning process can be avoided. We demonstrate the proposed method, Predictive Probabilistic Merging of Policies (PPMP), in combination with DDPG. In experiments on continuous control problems of the OpenAI Gym, we achieve drastic improvements in sample efficiency, final performance, and robustness to erroneous feedback, both for human and synthetic feedback. Additionally, we show solutions beyond the demonstrated knowledge.Comment: 6 page

Menopausia y síndrome metabólico

The rise in abdominal visceral fat and the increase in resistance to insulin in postmenopausal women are main factors in the development of metabolic syndrome X, defined as a group of clinical changes which include obesity, hyperglycaemic disorders, hypertension and dyslipidaemia. All of these disorders increase the risk of cardiovascular disease, one of the main causes of morbidity and mortality in women over the age of 55 in Western countries. Endogenous oestrogen seems to be a cardioprotector and the oestrogen deficiency that occurs during the menopause is associated with a number of metabolic changes which increase the cardiovascular risk. This work addresses the main topics related to metabolic syndrome X as well as its treatment and the impact it has on lifestyle during the menopause.El incremento en el contenido de grasa visceral abdominal y el aumento en la resistencia a insulina de la mujer post menopáusica son eventos principales en el desarrollo del síndrome metabólico; que se define como un grupo de alteraciones clínicas que incluye la obesidad, trastornos hiperglicemicos, hipertensión y dislipidemia1. Todos estos trastornos aumentan el riesgo de enfermedad cardiovascular, siendo una de las principales causas de morbi-mortalidad en las mujeres mayores de 55 años en los países occidentales. Los estrógenos endógenos parecen ser cardio protectores y la deficiencia estrogénica de la menopausia se asocia a un sin número de alteraciones metabólicas que incrementan el riesgo cardiovascular. En esta revisión se abordan los principales tópicos relacionados con el síndrome metabólico así como su tratamiento durante la menopausia y modificación del estilo de vida

Niveles de vitamina D en pacientes con osteoporosis en la ciudad de Neiva, Huila, Colombia

Objetivos: Determinar la prevalencia de los niveles bajos de vitamina D en pacientes con diagnóstico de osteoporosis por densitometría ósea en la ciudad de Neiva, Huila, Colombia. Materiales y métodos: Se incluyeron pacientes mayores de 50 años, durante el periodo comprendido entre el 01 de enero de 2011 hasta el 31 de diciembre de 2011, con diagnóstico de osteoporosis por densitometría y reporte de 25 OH hidroxivitamina D medida por quimioluminiscencia que consultaron al hospital universitario de la ciudad de Neiva. Se establecieron medidas de posible asociación entre variables cuantitativas (Odds Ratio) y se planteó un modelo de regresión lineal en el cual se determinó la variable independiente que mejor predice el resultado de la variable dependiente, para así poder observar la correlación existente. El análisis estadístico se realizó bajo el paquete SPSS versión 19. Resultados: La hipovitaminosis D es un trastorno muy frecuente en la población estudiada (89%) y en el 55% de los casos se acompaña de hiperparatiroidismo secundario. Se encontró una prevalencia de niveles de deficiencia del 35,5% (n= 20), de insuficiencia del 53,5% (n= 30) y óptimos del 11% (n= 6). Al realizar un análisis bivariado de la densidad mineral ósea (DMO) y los niveles de vitamina D, se observó que la DMO descendía simultáneamente con la caída en los niveles séricos de vitamina D. Esta asociación fue estadísticamente significativa a nivel de la columna lumbar p= 0,0063. Conclusiones: La insuficiencia y la deficiencia de vitamina D son muy frecuentes aun en zonas donde la exposición solar es diaria durante todo el año, lo que hace necesario realizar su determinación en todos los pacientes con osteoporosis. Abstract Objective: Determine the prevalence of vitamin D levels in patients diagnosed by bone densitometry for osteoporosis inNeiva, Huila, Colombia. Material and methods: Patients older than 50 years were included during the period from January 1, 2011 until December 31, 2011 , with a diagnosis of osteoporosis by densitometry and reporting of 25 -OH vitamin D by chemiluminescence, who attended the university hospital Neiva. Measures possible association between qualitative variables (Odds Ratio) is established and a linear regression model in which the independent variable that best predicts the outcome of the dependent variable, and observe the correlation was determined. Statistical analysis was performed on SPSS version 19 package. Results: Hypovitaminosis D is a very common disorder in the study population (89%) and in 55% of cases is associated with secondary hyperparathyroidism. Prevalence of deficiency levels of 35.5% (n = 20), insufficiency of 53.5% (n = 30) and optimal11% ( n = 6 ) were found. When performing a bivariate analysis of the levels of bone mineral density (BMD) and vitamin D levels, we found that bone mineral density down simultaneously with the fall in serum levels of vitamin D. This association was statistically significant at the level of lumbar spine p = 0.0063. Conclusion: Insufficiency and deficiency of vitamin D is very common even in areas where sun exposure is daily throughout the year, making it necessary to perform testing in all patients with osteoporosis

Interactive Imitation Learning in Robotics: A Survey

Interactive Imitation Learning (IIL) is a branch of Imitation Learning (IL) where human feedback is provided intermittently during robot execution allowing an online improvement of the robot's behavior. In recent years, IIL has increasingly started to carve out its own space as a promising data-driven alternative for solving complex robotic tasks. The advantages of IIL are its data-efficient, as the human feedback guides the robot directly towards an improved behavior, and its robustness, as the distribution mismatch between the teacher and learner trajectories is minimized by providing feedback directly over the learner's trajectories. Nevertheless, despite the opportunities that IIL presents, its terminology, structure, and applicability are not clear nor unified in the literature, slowing down its development and, therefore, the research of innovative formulations and discoveries. In this article, we attempt to facilitate research in IIL and lower entry barriers for new practitioners by providing a survey of the field that unifies and structures it. In addition, we aim to raise awareness of its potential, what has been accomplished and what are still open research questions. We organize the most relevant works in IIL in terms of human-robot interaction (i.e., types of feedback), interfaces (i.e., means of providing feedback), learning (i.e., models learned from feedback and function approximators), user experience (i.e., human perception about the learning process), applications, and benchmarks. Furthermore, we analyze similarities and differences between IIL and RL, providing a discussion on how the concepts offline, online, off-policy and on-policy learning should be transferred to IIL from the RL literature. We particularly focus on robotic applications in the real world and discuss their implications, limitations, and promising future areas of research

Encerado diagnóstico digital de los pacientes por parte de los alumnos de quinto de carrera. Estudio piloto

Con este proyecto nos proponemos introducir un aspecto básico del diagnóstico protésico y estético, el encerado digital. El objetivo final es proporcionar al alumno material de trabajo y herramientas que le permitan estudiar, aprender y practicar este procedimiento diagnóstico sin su presencia física en la facultad, o sin la necesidad de utilizar herramientas analógicas como se viene haciendo desde hace años.Depto. de Odontología Conservadora y PrótesisFac. de OdontologíaFALSEsubmitte

Knowledge- and ambiguity-aware robot learning from corrective and evaluative feedback

In order to deploy robots that could be adapted by non-expert users, interactive imitation learning (IIL) methods must be flexible regarding the interaction preferences of the teacher and avoid assumptions of perfect teachers (oracles), while considering they make mistakes influenced by diverse human factors. In this work, we propose an IIL method that improves the human–robot interaction for non-expert and imperfect teachers in two directions. First, uncertainty estimation is included to endow the agents with a lack of knowledge awareness (epistemic uncertainty) and demonstration ambiguity awareness (aleatoric uncertainty), such that the robot can request human input when it is deemed more necessary. Second, the proposed method enables the teachers to train with the flexibility of using corrective demonstrations, evaluative reinforcements, and implicit positive feedback. The experimental results show an improvement in learning convergence with respect to other learning methods when the agent learns from highly ambiguous teachers. Additionally, in a user study, it was found that the components of the proposed method improve the teaching experience and the data efficiency of the learning process.</p

An Interactive Framework for Learning Continuous Actions Policies Based on Corrective Feedback

© 2018, Springer Science+Business Media B.V., part of Springer Nature.The main goal of this article is to present COACH (COrrective Advice Communicated by Humans), a new learning framework that allows non-expert humans to advise an agent while it interacts with the environment in continuous action problems. The human feedback is given in the action domain as binary corrective signals (increase/decrease the current action magnitude), and COACH is able to adjust the amount of correction that a given action receives adaptively, taking state-dependent past feedback into consideration. COACH also manages the credit assignment problem that normally arises when actions in continuous time receive delayed corrections. The proposed framework is characterized and validated extensively using four well-known learning problems. The experimental analysis includes comparisons with other interactive learning frameworks, with classical reinforcement learning approaches, and with human teleoperators try